{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacked LSTMs for Time Series Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now build a slightly deeper model by stacking two LSTM layers using the Quandl stock price data (see the stacked_lstm_with_feature_embeddings notebook for implementation details). Furthermore, we will include features that are not sequential in nature, namely indicator variables for identifying the equity and the month."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run inside docker container for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [tensorflow guide](https://www.tensorflow.org/install/docker) and more detailed [instructions](https://blog.sicara.com/tensorflow-gpu-opencv-jupyter-docker-10705b6cd1d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`docker run -it -p 8889:8888 -v /path/to/machine-learning-for-trading/18_recurrent_neural_nets:/rnn --name tensorflow tensorflow/tensorflow:latest-gpu-py3 bash`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside docker container: \n",
    "`jupyter notebook --ip 0.0.0.0 --no-browser --allow-root`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from datetime import datetime, date\n",
    "from sklearn.metrics import mean_squared_error, roc_auc_score\n",
    "from sklearn.preprocessing import minmax_scale\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, LSTM, Input, concatenate, Embedding, Reshape\n",
    "import keras\n",
    "import keras.backend as K\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')\n",
    "np.random.seed(42)\n",
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data produced by the notebook [build_dataset](00_build_dataset.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 1167341 entries, 2009-01-01 to 2017-12-01\n",
      "Data columns (total 66 columns):\n",
      "ticker      1167341 non-null int64\n",
      "1           1167341 non-null float64\n",
      "2           1167341 non-null float64\n",
      "3           1167341 non-null float64\n",
      "4           1167341 non-null float64\n",
      "5           1167341 non-null float64\n",
      "6           1167341 non-null float64\n",
      "7           1167341 non-null float64\n",
      "8           1167341 non-null float64\n",
      "9           1167341 non-null float64\n",
      "10          1167341 non-null float64\n",
      "11          1167341 non-null float64\n",
      "12          1167341 non-null float64\n",
      "13          1167341 non-null float64\n",
      "14          1167341 non-null float64\n",
      "15          1167341 non-null float64\n",
      "16          1167341 non-null float64\n",
      "17          1167341 non-null float64\n",
      "18          1167341 non-null float64\n",
      "19          1167341 non-null float64\n",
      "20          1167341 non-null float64\n",
      "21          1167341 non-null float64\n",
      "22          1167341 non-null float64\n",
      "23          1167341 non-null float64\n",
      "24          1167341 non-null float64\n",
      "25          1167341 non-null float64\n",
      "26          1167341 non-null float64\n",
      "27          1167341 non-null float64\n",
      "28          1167341 non-null float64\n",
      "29          1167341 non-null float64\n",
      "30          1167341 non-null float64\n",
      "31          1167341 non-null float64\n",
      "32          1167341 non-null float64\n",
      "33          1167341 non-null float64\n",
      "34          1167341 non-null float64\n",
      "35          1167341 non-null float64\n",
      "36          1167341 non-null float64\n",
      "37          1167341 non-null float64\n",
      "38          1167341 non-null float64\n",
      "39          1167341 non-null float64\n",
      "40          1167341 non-null float64\n",
      "41          1167341 non-null float64\n",
      "42          1167341 non-null float64\n",
      "43          1167341 non-null float64\n",
      "44          1167341 non-null float64\n",
      "45          1167341 non-null float64\n",
      "46          1167341 non-null float64\n",
      "47          1167341 non-null float64\n",
      "48          1167341 non-null float64\n",
      "49          1167341 non-null float64\n",
      "50          1167341 non-null float64\n",
      "51          1167341 non-null float64\n",
      "52          1167341 non-null float64\n",
      "label       1167341 non-null int64\n",
      "month_1     1167341 non-null uint8\n",
      "month_2     1167341 non-null uint8\n",
      "month_3     1167341 non-null uint8\n",
      "month_4     1167341 non-null uint8\n",
      "month_5     1167341 non-null uint8\n",
      "month_6     1167341 non-null uint8\n",
      "month_7     1167341 non-null uint8\n",
      "month_8     1167341 non-null uint8\n",
      "month_9     1167341 non-null uint8\n",
      "month_10    1167341 non-null uint8\n",
      "month_11    1167341 non-null uint8\n",
      "month_12    1167341 non-null uint8\n",
      "dtypes: float64(52), int64(2), uint8(12)\n",
      "memory usage: 503.2 MB\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_hdf('data.h5', 'returns_weekly')\n",
    "data = data.drop([c for c in data.columns if str(c).startswith('year')], axis=1)\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train-test split\n",
    "\n",
    "To respect the time series nature of the data, we set aside the data at the end of the sample as hold-out or test set. More specifically, we'll use the data for 2018."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_size=52\n",
    "ticker = 1\n",
    "months = 12\n",
    "n_tickers = data.ticker.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = data[:'2016']\n",
    "test_data = data['2017']\n",
    "del data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each train and test dataset, we generate a list with three input arrays containing the return series, the stock ticker (converted to integer values), and the month (as an integer), as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([(1035424, 52, 1), (1035424,), (1035424, 12)], (1035424,))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train = [\n",
    "    train_data.loc[:, list(range(1, window_size+1))].values.reshape(-1, window_size , 1),\n",
    "    train_data.ticker,\n",
    "    train_data.filter(like='month')\n",
    "]\n",
    "y_train = train_data.label\n",
    "[x.shape for x in X_train], y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([(131917, 52, 1), (131917,), (131917, 12)], (131917,))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keep the last year for testing\n",
    "X_test = [\n",
    "    test_data.loc[:, list(range(1, window_size+1))].values.reshape(-1, window_size , 1),\n",
    "    test_data.ticker,\n",
    "    test_data.filter(like='month')\n",
    "]\n",
    "y_test = test_data.label\n",
    "[x.shape for x in X_test], y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def roc_auc(y_true, y_pred):\n",
    "    # any tensorflow metric\n",
    "    value, update_op = tf.metrics.auc(y_true, y_pred)\n",
    "\n",
    "    # find all variables created for this metric\n",
    "    metric_vars = [i for i in tf.local_variables() if 'auc_roc' in i.name.split('/')[1]]\n",
    "\n",
    "    # Add metric variables to GLOBAL_VARIABLES collection.\n",
    "    # They will be initialized for new session.\n",
    "    for v in metric_vars:\n",
    "        tf.add_to_collection(tf.GraphKeys.GLOBAL_VARIABLES, v)\n",
    "\n",
    "    # force to update metric values\n",
    "    with tf.control_dependencies([update_op]):\n",
    "        value = tf.identity(value)\n",
    "        return value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "# source: https://github.com/keras-team/keras/issues/3230\n",
    "def auc(y_true, y_pred):\n",
    "    ptas = tf.stack([binary_PTA(y_true, y_pred, k) for k in np.linspace(0, 1, 1000)], axis=0)\n",
    "    pfas = tf.stack([binary_PFA(y_true, y_pred, k) for k in np.linspace(0, 1, 1000)], axis=0)\n",
    "    pfas = tf.concat([tf.ones((1,)), pfas], axis=0)\n",
    "    binSizes = -(pfas[1:] - pfas[:-1])\n",
    "    s = ptas * binSizes\n",
    "    return K.sum(s, axis=0)\n",
    "\n",
    "\n",
    "def binary_PFA(y_true, y_pred, threshold=K.variable(value=0.5)):\n",
    "    \"\"\"prob false alert for binary classifier\"\"\"\n",
    "    y_pred = K.cast(y_pred >= threshold, 'float32')\n",
    "    # N = total number of negative labels\n",
    "    N = K.sum(1 - y_true)\n",
    "    # FP = total number of false alerts, alerts from the negative class labels\n",
    "    FP = K.sum(y_pred - y_pred * y_true)\n",
    "    return FP / (N + 1)\n",
    "\n",
    "\n",
    "def binary_PTA(y_true, y_pred, threshold=K.variable(value=0.5)):\n",
    "    \"\"\"prob true alerts for binary classifier\"\"\"\n",
    "    y_pred = K.cast(y_pred >= threshold, 'float32')\n",
    "    # P = total number of positive labels\n",
    "    P = K.sum(y_true)\n",
    "    # TP = total number of correct alerts, alerts from the positive class labels\n",
    "    TP = K.sum(y_pred * y_true)\n",
    "    return TP / (P + 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Define the Model Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The functional API of Keras makes it easy to design architectures with multiple inputs and outputs. This example illustrates a network with three inputs, as follows:\n",
    "\n",
    "- A two stacked LSTM layers with 25 and 10 units respectively\n",
    "- An embedding layer that learns a 10-dimensional real-valued representation of the equities\n",
    "- A one-hot encoded representation of the month\n",
    "\n",
    "This can be constructed using just a few lines - see e.g., \n",
    "- the [general Keras documentation](https://keras.io/getting-started/sequential-model-guide/), \n",
    "- the [LTSM documentation](https://keras.io/layers/recurrent/).\n",
    "\n",
    "Make sure you are initializing your optimizer given the [keras-recommended approach for RNNs](https://keras.io/optimizers/) \n",
    "\n",
    "We begin by defining the three inputs with their respective shapes, as described here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "returns = Input(shape=(window_size, n_features), name='Returns')\n",
    "tickers = Input(shape=(1,), name='Tickers')\n",
    "months = Input(shape=(12,), name='Months')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSTM Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To define stacked LSTM layers, we set the `return_sequences` keyword to `True`. This ensures that the first layer produces an output that conforms to the expected three-dimensional input format. Note that we also use dropout regularization and how the functional API passes the tensor outputs from one layer to the subsequent layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm1_units = 25\n",
    "lstm2_units = 10\n",
    "n_features = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm1 = LSTM(units=lstm1_units, \n",
    "             input_shape=(window_size, n_features), \n",
    "             name='LSTM1', \n",
    "             dropout=.2,\n",
    "             return_sequences=True)(returns)\n",
    "lstm_model = LSTM(units=lstm2_units, \n",
    "             dropout=.2,\n",
    "             name='LSTM2')(lstm1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding layer requires the `input_dim` keyword, which defines how many embeddings the layer will learn, the `output_dim` keyword, which defines the size of the embedding, and the `input_length` keyword to set the number of elements passed to the layer (here only one ticker per sample). \n",
    "\n",
    "To combine the embedding layer with the LSTM layer and the months input, we need to reshape (or flatten) it, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    }
   ],
   "source": [
    "ticker_embedding = Embedding(input_dim=n_tickers, \n",
    "                             output_dim=10, \n",
    "                             input_length=1)(tickers)\n",
    "ticker_embedding = Reshape(target_shape=(10,))(ticker_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concatenate Model components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can concatenate the three tensors and add fully-connected layers to learn a mapping from these learned time series, ticker, and month indicators to the outcome, a positive or negative return in the following week, as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged = concatenate([lstm_model, ticker_embedding, months], name='Merged')\n",
    "hidden_dense = Dense(10, name='FC1')(merged)\n",
    "output = Dense(1, name='Output')(hidden_dense)\n",
    "\n",
    "rnn = Model(inputs=[returns, tickers, months], outputs=output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary lays out this slightly more sophisticated architecture with 29,371 parameters, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Returns (InputLayer)            (None, 52, 1)        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Tickers (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "LSTM1 (LSTM)                    (None, 52, 25)       2700        Returns[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1, 10)        24890       Tickers[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "LSTM2 (LSTM)                    (None, 10)           1440        LSTM1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "reshape_1 (Reshape)             (None, 10)           0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "Months (InputLayer)             (None, 12)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Merged (Concatenate)            (None, 32)           0           LSTM2[0][0]                      \n",
      "                                                                 reshape_1[0][0]                  \n",
      "                                                                 Months[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "FC1 (Dense)                     (None, 10)           330         Merged[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "Output (Dense)                  (None, 1)            11          FC1[0][0]                        \n",
      "==================================================================================================\n",
      "Total params: 29,371\n",
      "Trainable params: 29,371\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "rnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compile the model to compute a custom auc metric as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn.compile(loss='binary_crossentropy', \n",
    "            optimizer='adam',\n",
    "           metrics=['accuracy', auc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_path = 'models/quandl.lstm_months_{}_{}.weights.best.hdf5'.format(lstm1_units, lstm2_units)\n",
    "checkpointer = ModelCheckpoint(filepath=rnn_path,\n",
    "                              monitor='val_loss',\n",
    "                              save_best_only=True,\n",
    "                              save_weights_only=True,\n",
    "                              period=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping = EarlyStopping(monitor='val_loss', \n",
    "                              patience=5,\n",
    "                              restore_best_weights=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "result = rnn.fit(X_train,\n",
    "                 y_train,\n",
    "                 epochs=50,\n",
    "                 batch_size=32,\n",
    "                 validation_data=(X_test, y_test),\n",
    "                 callbacks=[checkpointer, early_stopping],\n",
    "                 verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training stops after 18 epochs, producing a test area under the curve (AUC) of 0.63 for the best model with 13 rounds of training (each of which takes around an hour on a single GPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_history = pd.DataFrame(result.history)\n",
    "loss_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def which_metric(m):\n",
    "    return m.split('_')[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_history.groupby(which_metric, axis=1).plot(figsize=(14, 6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict = pd.Series(rnn.predict(X_test).squeeze(), index=y_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roc_auc_score(y_score=test_predict, y_true=y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn.load_weights(rnn_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict = pd.Series(rnn.predict(X_test).squeeze(), index=y_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roc_auc_score(y_score=test_predict, y_true=y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = (test_predict.to_frame('prediction').assign(data='test')\n",
    "               .append(train_predict.to_frame('prediction').assign(data='train')))\n",
    "predictions.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = sp500_scaled.join(predictions).dropna()\n",
    "results.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = {}\n",
    "for run, df in results.groupby('data'):\n",
    "    corr[run] = df.SP500.corr(df.prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp500_scaled['Train Prediction'] = pd.Series(train_predict.squeeze(), index=y_train.index)\n",
    "sp500_scaled['Test Prediction'] = pd.Series(test_predict.squeeze(), index=y_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_error = np.sqrt(rnn.evaluate(X_train, y_train, verbose=0))\n",
    "testing_error = np.sqrt(rnn.evaluate(X_test, y_test, verbose=0))\n",
    "print('Training Error: {:.4f} | Test Error: {:.4f}'.format(training_error, testing_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
